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ABSTRACT 

Changes that have been made in reporting and 
disseminating information from the National Assessment of Educational 
Progress are discussed, focusing on those changes brought about by 
the first Trial State Assessment (TSA) . The TSA collected data on the 
mathematics performance of students in grades 4, 8, and 12 in public 
and private schools in 40 states. This voluminous material was not 
well-digested by the media or the public. In contrast, briefer and 
more focused reports on science achievement and writing achievement 
were more enthusiastically received and more readily used. The 
National Center for Education Statistics (NCES) then developed a 
two-part plan for data release. Part 1 involved data release in 
smaller portions, and part 2 consisted of improvement of the data 
presentation. These improvements were initiated with a conference of 
measurement professionals who planned new comparison charts and 
better ways of formatting the information. Five figures illustrate 
some of the new approaches. (SLD) 
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In June 1991 , the National Assessment of Educational Progress 
(NAEP) released results from the first ever Trial State Assessment. 
Data were released on the performance of eighth grade public school 
students in 40 states and the national data were released on 
students in grades 4, 8, and 12 in both public and private schools. 
There was a mountain of information on how and what students learn 
in mathematics . The report was 530 pages long and over 1 1/4" 
thick. We released data on the traditional NAEP reporting 
categories for the nation and the states , which included average 
proficiency by race/ethnicity, gender, type of community, region, 
type of school, and parents' highest level of education. Along 
with this we reported: 

proficiencies in various content areas and detailed 
descriptions of students' mathematical abilities, 
course taking patterns (vital to an understanding of math 
proficiencies ) 

students performance on constructed-response questions in 

various content areas 
instructional approaches such as ability grouping and the 
use of instructional materials 

students use of computers and calculators in school 
instructional time and emphasis 

students perceptions and personal experiences, and 
information on mathematics teachers 

We thought this information would keep researchers, policymakers, 
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and educators busy for several years. But, we were disappointed in 
the use that was made of this voluminous material. Reporters 
concentrated on the horse-race aspects of the Trial State 
Assessment, and researchers and policymakers moved on to the next 
report, rather than continuing to mine the data from this one. 
In fact, we were so busy getting the next series of reports ready 
that we didn't immediately look back either. 

The next reports came out in the Spring of 1992. NAEP released The 
Science Report Card, Reading In and Out of School, and Exploring 
New Methods for Collecting Students' School Based Writing. These 
reports were all about 100 pages long and focused on a very limited 
content area. Surprisingly, we received tremendous press coverage 
and intense interest from data users on all three of these reports. 
Pound for pound, these reports were better exploited than the 
comprehensive mathematics report. Less was more. 

Meanwhile, back at NCES we were in the process of collecting 
volumes of data for the 1992 TSA in mathematics and reading. We 
knew there was a demand for the data, but how could we improve its 
impact and make it more accessible? The volume of the 1990 TSA 
was simply too intimidating. We didn't want to repeat this mistake 
in 1992. 

As a result, we developed a two part plan. Part I was a plan for 
the release of the data. Rather than put out volumes of 
information all at once, we decided to release smaller, more 
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focused reports , scattered across several months. The first 
release of tae TSA in mathematics would concentrate on proficiency 
and achievement levels in the traditional NAEP reporting 
categories. And because we were adding achievement levels , where 
previously we had reported at anchor points using scale scores, we 
decided to also release a supplemental report that would explain 
some of the ways that NAEP scores can be interpreted. This release 
would answer the question of "how US students are performing in 
mathematics" and how to judge that performance. 

Subsequently , we would release smaller , more focused reports on 
critical aspects of mathematics education that NAEP collected. 
Several reports were planned: one on students' opportunities to 
learn mathematics — course offerings and content. Since there is 
so much interest in alternative assessments and NAEP has been at 
the forefront of this effort, a second focused report was planned 
to discuss students' responses to constructed response items. We 
also planned to release the reading results in a similar manner 
with focused reports on the supplemental material. 

Part I of the strategy was to phase the release of information to 
concentrate attention on the substance of the data collections. 
Part II of the strategy was to look into how to improve the 
presentation of the data. In the summer of 1992 we, therefore, 
called together representatives from a wide range of organizations 
for brainstorming sessions on how to improve the NAEP reporting 
formats. Rather than a homogeneous group, we called together 
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Average Overall Mathematics Proficiency by Selected 
Demographics for Five Performance Bands (Quintiles) 
1992 Grade 8 V ' 
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Average Proficiency by Mathematics Subscales for Five 
Performance Bands (Quintiles) 
1992 Grades 4 and 8 
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representatives from other federal agencies, researchers, 
educators, teachers, state representatives, and education 
associations. The results were stimulating. These groups suggested 
innovative ideas on how to improve our reporting and formatting. 
Eugene Johnson from ETS was present at one of the meetings, and as 
the group mulled through a new idea on how to report the data, he 
doodled away. By the next day when he was back in Princeton, he 
had started to implement one of these ideas. He was responding to 
a suggestion by the group that a consumer report type chart that 
would show states proficiencies at delivering different educational 
variables could be useful. With such a chart you would be able to 
look at average mathematics proficiency in the five content areas, 
or how different types of students perform in the states relative 
to the same types of students in other states. 

(Table 2.1) For example, Colorado does very well on average, but 
less well for advantaged urban students. On the other hand, 
California is in the top 20 percent for its advantaged urban 
students, but lower in lower categories for students in 
disadvantaged urban or rual areas. 

(Table 7) Similarly, you can see that most states are relatively 
consistent in the delivery of mathematics across the five content 
areas, but the chart helps you readily visualize the states where 
delivery is not consistent. For example, California seems to do 
relatively well in geometry as compared with the performance in 
numbers and operations , measurement , data analy s is , algebra , and 

ERIC 8 



0 

ERIC 



FIGURE 1 



Comparisons of Overall Mathematics Average Proficiencv 
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□ 
□ 



State has statistically significantly higher average 
proficiency than the state listed at the top of the chart. 

No statistically significant difference from the state 
listed at the top of the chart. 

State has statistcallv significantly lower average 
proficiency than the state listed a: the top of the chart. 
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The betveen state comparisons take into account sampling and 
measurement error and that each state is being compared with 
every other state. Significance is determined by an application of 
the Bonfen-oni procedure based on 946 comparisons by 
comparing the difference between the two means with four times 
the square root of the sum of the squared standard errors. 



FIGURE 2 



Distribution of Overall Mathematics Proficiency Organized by 
Average Proficiency 
1992 Grade 4 
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The center darkest box indicates a simultaneous confidence interval around the 
average rnauiemasics proficiency for the state based oc the Bonferroni procedure 
for multiple comparisons. Center boxes that do not overlap indicate significant 
differnces between between stales in average mathemaiics proficiency. The 
darter shaded boxes indicate the ranges between the 25th and 75th percentiles 
of the mathematics proficiency distribution, and the tighter shaded boxes the 
ranges between the 5th to 25th percentiles and the 75th to 95th percentiles of the 
distribution. 
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estimation in that state. 



We talked at length about the Bonferroni multiple comparison chart , 
and the consensus among our data users was that it was necessary , 
accurate , we had used it before and should stay with it. People 
would get used to it. However , to make it more readable across the 
lines , we added the state letters. 

These groups also suggested a map showing the same information as 
contained in the multiple comparison chart. ETS developed two maps 
to illustrate these differences among states. One map for each 
state shows in which states performance is the same, better than, 
or worse than, the state that is the focus for the map. Another 
chart shows changes from 1990 and 1992. These are simply different 
ways of illustrating information that was already available , but we 
think that they make the information much more accessible and 
understandable to more people „ 

There were many more suggestions for improving the reporting and 
formatting of information from the National Assessment of 
Educational Progress. Some are excellent , like the request for a 
directory of where to find NAEP information in the many reports , 
but we don't have the resources at the moment to implement all of 
them. We have made every effort to improve not only the NAEP data, 
but its usefulness to its expanding audience of data users. We 
welcome any more suggestions or comments you might want to offer. 
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